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Foreward  (This  foreward  is  not  part  of  the  Profile) 

This  profile  defines  functional  requirements  for  an  interoperable  Fibre  Channel  based 
instrumentation  communications  bus. 

This  profile  was  prepared  by  the  NexGenBus  project  team.  The  project  was  started  in  1997. 

Requests  for  interpretation,  suggestions  for  improvements  or  addenda,  or  defect  reports  are 
welcome.  They  should  be  sent  to  the  NexGenBus  project  office,  47758  Ranch  Road,  Bldg  1492, 
Unit  1,  Patuxent  River,  MD  20670-1456  or  JonesSR@Navair.Navy.Mil.  Additional  information 
may  be  found  at  http://nexgenbus.nawcad.navy.mil. 

The  initial  draft  will  be  released  for  general  comment  with  a  presentation  made  at  ITC/USA  ’99 
in  Las  Vegas,  October  26-28.  Comments  from  the  initial  draft  will  be  incorporated  and 
submitted  to  the  Range  Commanders  Council  (RCC)  Telemetry  Group.  The  RCC  will  initiate 
the  formal  Pink  Sheet  process  where  comments  will  be  solicited  before  becoming  an  official 
RCC  standard. 

The  members  of  the  NexGenBus  team  at  the  time  of  this  draft  ark  M  y 

Ai  .ih  ii  | 

Organization  Represented  Name 

Aberdeen  Proving  Ground  /  Mardemess 

Eagan,  McAllister,  and  Associates  'J  // T<j>ni  DeSelms 

Eagan,  McAllister,  and  Associates  ,  \  •  -J  ‘V//4Dom  Garuccio 

Edwards  Air  Force  Base  /  .  Kip  Temple 

Naval  Air  Warfare  Center  -  Aircraft  Division  | :  j  {  Thomas  Grace 
Naval  Air  Warfare  Center  -  Aircraft  Division  -  Sid  Jones 

Naval  Air  Warfare  Center  -  Aircraft  Division  Mark  Smedley 
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1  Introduction 


1.1  Purpose 

This  Instrumentation  Profile  is  intended  to  provide  a  starting  point  for  interoperability  of  Fibre 
Channel  end-items  in  a  test-vehicle  instrumentation  environment.  It  is  envisioned  this  profile 
will  be  one  of  a  family  of  interoperability  documents.  When  taken  as  a  whole,  interoperability 
between  compliant  nodes  will  be  assured.  Since  this  document  is  focused  at  the  system  level,  the 
target  audience  is  both  the  end-item  designer  concerned  about  interoperability  and  the 
instrumentation  engineer  concerned  with  understanding  the  capabilities  and  tradeoffs  of  such  a 
system. 


1.2  Scope 

This  document  specifies  a  minimum  required  to  achieve  interoperability  between  multiple 
vendors’  end-items  on  a  Fibre  Channel  instrumentation  bus.  This  docunient  only  addresses  the 
ability  to  move  the  data.  The  format  of  the  data  is  beyond  the  scope  of  this  document. 

1.3  Document  Structure  j  jt  ! 

Section  1  provides  top  level  information  to  help  the  reader  understand  how  to  get  the  most  from 
this  document.  /%y  j  ;  j  ! 

Section  2  lists  other  references  important  for  a  complete  understanding 
Section  3  explains  new  terms  and  abbreviations  <  A  ;|  [■ 

Section  4  addresses  interoperability  issues  that  may  affect  an  instrumentation  system  network. 
Section  5  is  normative  and  addresses  the  issues  defined  in  section  four  against  the  relative 
standards  where  interoperability  would  beimpacted. 

Section  6  is  informative  and  covers  many  issues  that  may  make  the  system  more  usable,  capable, 
or  fault  tolerant,  but  are  not  specifically  required  for  interoperability. 


1.4  Precedence 

The  order  of  precedence  for  instrumentation  interoperability  shall  be  this  document,  the  FC-AE 
profile  (when  published),  and  the  Fibre  Channel  suite  of  standards. 


1.5  Responsibility 

This  document  is  a  result  of  a  joint  effort  between  the  Office  of  the  Secretary  of  Defense  (OSD) 
Central  Test  &  Evaluation  Program  (CTEIP)  Office  and  the  Range  Commanders  Council  (RCC) 
Telemetry  Group.  Cognizance  of  this  profile  remains  with  the  RCC  Telemetry  Group. 


The  Fibre  Channel  documents  referenced  throughout  this  profile  are  the  responsibility  of  the  Til 
Technical  Committee  (TC)  under  Accredited  Standards  Committee  (ASC)  NCITS  (National 
Committee  for  Information  Technology  Standardization).  In  turn,  NCITS  operates  under  the 
procedures  of  the  American  National  Standards  Institute  (ANSI). 
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1.6  Approach 

Interoperability  and  system  requirements  defined  early  in  the  Next  Generation  Instrumentation 
Bus  Project  were  used  as  a  baseline.  The  system  requirements  were  identified  as  either  interface 
or  application  requirements.  The  interface  requirements  are  necessary  for  two  nodes  within  a 
network  to  exchange  data.  Interface  requirements  are  considered  normative  and  are  the  basis  for 
sections  4  and  5.  Application  requirements  are  necessary  for  an  instrumentation  network  to 
perform  in  a  test  vehicle  environment,  but  may  not  be  needed  for  interoperability.  Application 
requirements  may  be  normative  and  included  in  section  5  or  they  may  be  informative  and  added 
to  section  6. 

The  Fibre  Channel  Avionics  Environment  (FC-AE)  sub-committee  of  Technical  Committee  T1 1 
is  working  on  a  similar  document  for  production  avionics  use.  The  required/desired  deviations 
to  the  Fibre  Channel  Standards  for  avionics  applications  are  worked  through  T1 1  by  the  FC-AE. 
Requirements  for  avionics  applications  are  practically  the  same  as  for  instrumentation 
applications.  The  major  issues  the  instrumentation  community  may  raise  with  Fibre  Channel  are 
expected  to  be  resolved  by  the  FC-AE  due  to  the  similarities.  In  the  cas|where  the 
instrumentation  environment  may  have  a  greater  need,  the  issue  will  be  'worked  through  the  FC- 
AE  or  other  T1 1  committees  as  appropriate. 


2  References 

2.1  Standards 

ANSI  X3. 230- 1994 

ANSI  X3. 297- 1997 

ANSI  X3. 303- 1998 

ANSI  X3. 272- 1996 

ANSI  X3.nnn- 1999 
[ANSI  X3.nnn-yyyy 
[RFC  791 
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Information  Technology  -  Fibre  Channel  Physical  and  Signaling 
Interface  (FC-PH),  1994 

Information  Technology  -  Fibre  Channel  Physical  and  Signaling 
Interface  -  2  (FC-PH-2),  1997 

Information  Technology  -  Fibre  Channel  Physical  and  Signaling 
Interface  -  3  (FC-PH-3),  1998 

Information  Technology  -  Fibre  Channel  Arbitrated  Loop  (FC-AL), 
1996 

Fibre  Channel  Avionics  Environment  Technical  Report  (due  12/99) 
FC-IP? ] 

DARPA  Internet  Program  Protocol  Specification,  September,  1981] 


1  / 

:  > 
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3  Abbreviations,  Definitions,  &  Conventions 

Arbitrated  Loop  -  A  Fibre  Channel  topology  where  nodes  are  linked  together  in  a  closed  loop. 

Traffic  is  managed  with  a  token-acquisition  protocol,  and  only  one  connection  can 
be  maintained  in  the  loop  at  a  time. 

Class  1  -  Dedicated  connection  allocating  full  bandwidth  between  a  pair  of  ports.  Class  1 
provides  confirmation  of  delivery  or  notification  of  non-delivery  between  the 
source  and  destination  ports. 

Class  2  -  Connectionless  class  of  service  with  confirmation  of  delivery  or  notification  of  non- 
deliverability  of  frames.  No  bandwidth  is  allocated  or  guaranteed. 

Class  3  -  Connectionless  class  of  service  providing  a  datagram-like  delivery  service  with  no 
confirmation  of  delivery,  or  notification  of  non-delivery. 

Class  4  -  Connection  oriented  class  of  service  which  provides  a  virtual  circuit  between  a  pair  of 
ports  with  guaranteed  bandwidth  and  latency  with  confirmation  of  delivery  and 
notification  of  non-delivery. 

Class  6  -  A  derivative  of  class  1  that  provides  a  reliable  one  to  many  multicast  service  with 
confirmation  of  delivery  and  notification  of  non-deliver^, 
classes  of  service 

command-response  architecture  -  A  network  which  contains  q  device  which  controls  the 
access  of  the  other  nodes  to  the  network.  1 

counter-rotating  ring  -  An  arrangement  whereby  two  signal  piaths[  the  directions  of  which  are 
opposite,  exist  in  a  physical  ring  or  loop  topology.  \ 

F_Port  -  Fabric  Port.  A  Fibre  Channel  term,  referring  to.the  port  residing  on  the  Fabric 

(Switch)  side  of  the  lirilc.  It  attaches  to  a  NJPort  (Node  Port)  at  the  connected 
device,  across  a  link.  "N  \  i  /  (J- 
fabric  -  denotes  the  interconnect  of  ports  jwithout  regard  to  topology 

Fabric  -  The  Fabric  is  a  transport  medium  that  provides  switched  interconnects  between  ports. 

Fabric  specifies  a  topology  distinct  from  Point-to-Point  and  Arbitrated  Loop, 
informative  -  Information  provided  for  completeness.  Not  required, 
interoperability  -  The  capability  to  communicate  or  transfer  data  among  various  functional 
units  in  a  manner  that  requires  the  user  to  have  little  or  no  knowledge  of  the 
unique  characteristics  of  those  units. 

Internet  Protocol  (IP)  -  Part  of  the  TCP/IP  family  of  protocols  describing  software  that  tracks 
the  Internet  address  of  nodes,  routes  outgoing  messages,  and  recognizes  incoming 
messages. 

N_Port  -  Node  Port.  A  Fibre  Channel  term,  referring  to  the  link  control  facility  which  connects 
across  a  link  to  the  F_Port  (Fabric  Port)  at  the  Fabric  (Switch). 

NL_Port 

node  -  A  point  of  connection  into  a  network.  In  Fibre  Channel,  a  collection  of  one  or  more 
N_Ports. 

normative  -  Required  for  compliance  to  prescribed  norms  or  standards, 
open  systems  -  Everyone  would  comply  with  a  set  of  hardware  and  software  standards, 
peer-to-peer  architecture  -  A  network  that  contains  equivalent  nodes  with  respect  to  their 
capability  of  control  or  operation. 

Point-to-Point  -  Fibre  Channel  topology  in  which  communication  between  two  N_Ports  occurs 
without  the  use  of  Fabric. 
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port  -  Network  access  point  for  data  entry  or  exit.  In  Fibre  Channel,  a  generic  reference  to  an 
N_Port  or  F_Port. 

protocol  -  A  procedure  for  adding  order  to  the  exchange  of  data.  A  specific  set  of  rules, 

procedures,  or  conventions  relating  to  format  and  timing  of  data  transmission 
between  two  devices. 

simultaneous  sampling  -  Acquiring  multiple  data  within  a  given  time  period. 

time  correlation  -  The  ability  to  relate  two  or  more  asynchronous  sources. 

time  synchronization  -  The  ability  to  synchronize  two  or  more  asynchronous  sources. 
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4  Transport-Level  Interoperability  Issues 

This  section  addresses  the  major  issues  affecting  transport  interoperability  of  an  instrumentation 
system  network.  Node  and  system  designers  should  be  aware  of  these  issues  and  their  effect  on  a 
given  implementation  when  beginning  a  new  design  or  modifying  an  existing  one.  Section  0 
addresses  the  document  structure  and  the  relationship  between  sections.  As  such,  each  of  the 
following  subsections  drives  changes  or  deviations  to  the  Fibre  Channel  standards. 

4. 1  Fibre  Channel  Level  0  (FC-0) 

The  FC-0  level  defines  the  physical  portions  of  Fibre  Channel,  including  the  media  types,  connectors,  and 
the  electrical  and  optical  characteristics  needed  to  connect  ports.  The  FC-0  level  is  designed  for 
maximum  flexibility.  It  allows  the  use  of  a  large  number  of  technologies  to  meet  the  widest  range  of 
system  requirements. 

4.1.1  Cables  and  Connectors 

In  a  military  test  vehicle,  the  environment  is  typically  harsh.  Commercial:  grade  cables  and 
connectors  can  be  degraded  through  vibration,  temperature,  and  altitude  to  the  point  of  failure. 
Cables  and  connectors  must  be  used  which  can  withstand  these  environments  without  sacrificing 
the  integrity  of  the  instrumentation  system.  ^  j 

4.1.2  Signaling  Rate  '  I 

Fibre  Channel  supports  several  signaling  rates  including  quarter  speed,  half  speed,  and  full 
speed.  There  is  work  within  the  Fibre  Channel  committees  to  allow  double  and  quadruple  speed 
in  future  revisions  of  the  standard.  A  fabric  may  operate  with  multiple  speeds  between  multiple 
ports.  However,  for  two  nodes  to  communicate;  they  must  be  operating  at  the  same  signaling 

ra,e'  i:  I  i  f  Gy 

4.1.3  Signal  Quality 

In  order  to  ensure  interoperability  between  implementations,  it  is  necessary  to  specify  the  signal 
characteristics  of  a  Fibre  Channel  transmitter.  Though  deviations  are  not  expected,  it  is 
important  any  cables  and/or  connectors  chosen  as  a  result  of  4.1.1  should  meet  similar 
characteristics. 

4.2  Fibre  Channel  Level  1  (FC-1) 

FC-1  defines  the  transmission  protocol.  Fibre  Channel  transmits  information  using  an  adaptive  8B/10B 
code  to  bound  the  maximum  run  length  of  the  code,  maintain  DC-balance,  and  provide  word  alignment. 

4.3  Fibre  Channel  Level  2  (FC-2) 

The  FC-2  level  defines  the  signaling  and  framing  protocol,  including  frame  layout,  frame  header  content, 
and  rules  for  use.  The  transported  data  is  transparent  to  FC-2  and  visible  to  FC-3  and  above. 

4.3.1  Port  Type 

Fibre  Channel  has  three  basic  topologies  as  stated  above.  These  three  topologies  require  two 
different  port  types.  Point-to-Point  and  Fabric  topologies  use  N_Ports  while  the  Arbitrated  Loop 
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topology  uses  NL_Ports.  [Is  an  NL_Port  a  superset  of  an  NJPort?  Will  an  NL_Port  work  in 
all  three  topologies?] 

4.3.2  Login 

Fibre  Channel  nodes  must  log-in  to  the  fabric  or  loop  (loop  initialization)  and  log-in  to  a  port 
before  data  can  be  exchanged  with  that  port.  This  process  allows  the  two  nodes  to  establish  their 
operating  environment. 

4.3.3  Class  of  Service 

Fibre  Channel  allows  several  methodologies  in  which  the  communication  circuit  is  allocated  and 
retained  by  the  communicating  N_Ports  and  by  the  level  of  the  delivery  integrity  required  for  an 
application.  These  methodologies  are  called  classes  of  service  and  are  denoted  by  a  class 
number.  Currently  Fibre  Channel  has  five  classes  of  service  as  listed  below. 


Service 
Class  1 
Class  2 
Class  3 
Class  4 
Class  6 


Description 


Dedicated  connection 


Multiplexed  connection 
Datagram 

Fractional  bandwidth 
Uni-directional  dedicated  connection 


4.4  Fibre  Channel  Level  3  (FC-3)  7  7.X 

FC-3  defines  the  common  services  that  may  be  available  across  multiple  ports  in  a  node. 

f  ^  J  r :■'{  f  i)d\  v 

4.5  Fibre  Channel  Level  4  (FC-4) 

FC-4  defines  the  mapping  between  loVver  levels  of  Fibre  Channel  and  the  command  sets  that  use  Fibre 
Channel.  !  7|  h  7  j;  j 


4.5.1  Protocol 

There  are  many  upper  layer  protocols  available  to  place  on  Fibre  Channel.  Fibre  Channel  allows 
multiple  protocols  on  the  network  concurrently.  However,  the  protocol  must  be  common  to 
communicating  ports.  Each  protocol  is  tuned  for  a  particular  application.  It  is  up  to  the  designer 
of  each  node  to  utilize  the  protocol(s)  that  is  (are)  best  suited  for  the  intended  purpose.  The 
system  designer  must  consider  the  protocols  available  on  each  node  when  designing  a  system. 
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5  Fibre  Channel  Deviations  and  Clarifications 

The  following  sub-sections  describe  the  mandatory  changes  to  the  indicated  standards  or  reports. 
The  majority  of  the  changes  are  concerned  with  making  optional  capabilities  mandatory  in  order 
to  increase  the  chances  of  interoperability. 


5. 1  Fibre  Channel  Physical  and  Signaling  Interface  (FC-PH-x) 

FC-PH-x  (X3.230.  X3.297,  X3.303) 

Section  Change _ _ _ _ 


Normative  References 


MIL-C-38999  Connectors,  Electrical,  Circular,  General  Specification  For. 
[Gore  connector  is  not  strictly  per  std,  Thomas  will  modify  appropriately] 


MIL-C- 17/Quad  Cable 
[Gore  cable  is  not  strictly  per  std ,  Thomas  mil  modify  appropriately] 


Definitions  and  Conventions 


3.1.70 


NL_Port  functionality  shall  be  required 


FC-0  Functional  Characteristics 


5.1 


Addition  of  Gore  cable  in  the  general  characteristic  section 


5.1 


1,063  Mbaud  support  required _ _ _ _ _ 

Media  designation  for  Quad  cable  will  be  ‘QU’  [We  need  to  pith  a  designation  to 
identify  this  quad  cable  in  the  FC-0  nometjcl^ire  Hke  in  Table  3  below.] 

Update  Table  3 


5.7 


5.8 


/Wii 


IV 


Part  of  Table  3,  Electrical  Media  Signal  Interface 
Overview 

100  MB/sec  1.062  5  Gbaud 

1 00-TV-EL-S 

100-MI-EL-S 

100-QU-EL-S 

Subclause  7.2 

Subclause  7.2 

Subclause  7.4 

0-25m 

0-1 0m 

0-25m 

Electrical  Cable  Interface  Specification 


[ Thomas^  updating  based_  on  lab  tests] 


[Update  table  10] _ _ _ 

Quad  Data  Link  --  Info  will  have  to  be  added  to  include  the  Gore  cable.  It  should 
follow  the  format  in  the  previous/current  sections.  Content  will  be  based  on  the 
results  from  the  test  plan  and  cable  mfr. 


7.4 


Electrical  Cable  Plant  Specification 


[Thomas^  updating  based  on  lab  tests[ _ _ _ _ _ 

Quad  Cable  Plant  Specification  (new  section)  A  new  section  will  have  to  be  added 
to  include  the  Gore  cable.  It  should  follow  the  format  in  the  previous  sections. 
Content  will  be  based  on  the  results  from  the  test  plan. _ 


9.5 


22 


Classes  of  Service 


22.3 


Class  3  -  Datagram  support  is  required. 
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FC-PH-x  ( X3.230 .  X3.297 .  X3.303) 

Section  Change 

23 

Login  and  Service  Parameters 

23 

Nodes  shall  support  implicit  login  and  optionally  support  explicit  login. 

{Here’s  my  thinking... see  section  6.3.4] 

5.2  Fibre  Channel  Arbitrated  Loop:  FC-AL 


FC-AL  (X3.272) 

Section 

Change 

11 

Clock  Synchronization  Service  (New  Section) 

Each  L_Port  shall  be  capable  of  storing  a  time  propagation  delay  value.  Whenever 
the  timeserver  sends  a  time  value,  the  L_Port  will  add  its  delay  value  to  the  time 
value  to  update  its  real-time  clock.  The  delay  value  format  shill  be  a  binary 
representation  of  nanoseconds  delay.  In  order  to  accommodate  the  maximum  delay 
from  a  timeserver,  a  16  bit  data  field  should  be  used.  /  7  j  t 

i  .;  ■;■!!  ■  .  j 

Max  delay  =  125  nodes  X 240ns  delay/node  +  126  links  XSns/m  X  30m  =  48,900ns 

L. 

i  M  •  :M 

/  •  >,  j 

5.3  FC-4  Upper  Layer  Protocols  1/  /Ji  j 

I-.'  •  •  ■  1 

;  1  V:  ^ 

,  i  v 

[Which  protocol  should  we  select ?\  Should  we  even  select  qne?]_ 


FC-IP.  RFC  791?  i  1  i  :i  I  Vv' 

Section  Change  1  i  1  l 

9 

• 

9  ]•  •;.-y  ^ 

IP  support  as  an  upper  ljayer  protocol  is  required 
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6  Instrumentation  System  Issues  (Informative) 

This  section  is  to  provide  insight  to  ideas,  both  good  and  bad,  that  may  affect  a  Fibre  Channel 
instrumentation  system.  As  such  requirements  are  not  to  be  construed. 

6.1  Architecture 

Fibre  Channel  by  itself  does  not  imply  the  type  of  architecture  an  instrumentation  system  must 
utilize.  There  are  two  basic  architectures  that  can  be  employed  in  the  design  of  the  system.  The 
nodes  may  or  may  not  support  both  architectures.  In  the  traditional  system,  a  controller  or 
master  is  used  to  command  the  nodes  and  receive  the  responses.  The  controller  is  programmed 
with  the  knowledge  of  the  overall  format  and  directs  each  node  to  acquire  data  and  respond 
(reference  Figure  1).  The  controller  typically  becomes  the  aggregator  of  the  data  as  it  formats 
the  output(s)  for  recording,  transmitting,  or  processing.  This  keeps  the  nodes  simple.  Traffic  on 
the  bus  is  very  orderly  based  on  what  the  controller  requests.  This  is  known  as  a  command- 
response  architecture.  Multiple  formats  can  be  stored  in  the  controller  and  changed  via  a  cockpit 
switch  or  sophisticated  uplink.  Controllers  can  vary  from  small,  inexpensive  units  that  are 
inflexible  to  large  expensive  units  that  can  do  everything.  /  7  :k 


Figure  1,  Controller  Based  Architecture 


Another  architecture  available  to  the  instrumentation  network  is  the  peer-to-peer  architecture. 
Each  node  is  programmed  with  its  own  schedule.  Individually  the  nodes  determine  when  to 
acquire  the  data,  what  format  to  packet  the  data  into,  and  to  whom  to  send  it  (reference  Figure  2). 
One  of  the  advantages  of  an  autonomous  system  is  the  ease  to  add  new  nodes.  Additional  nodes 
just  need  to  be  physically  connected  to  the  bus  and  programmed.  The  other  nodes  are  not 
affected  (assuming  plenty  of  bandwidth  on  the  bus  and  the  data  sinks).  One  node  could  still 
receive  all  the  data  and  format  it  into  the  proper  outputs  for  recording  and  transmitting  similar  to 
the  command  response  architecture. 
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Figure  2,  Peer-to-Peer  Architecture 

6.2  Open  System  Q  .  ) 

In  an  open  system,  the  specifications  are  generally  in  the  public^orriMn.  Of  particular 
importance  is  the  specifications  should  be  in  wide  use  as  well.  This  allows  ready  access  not  only 
to  the  specifications,  but  also  to  the  chipsets,  OEM  boards,  drivers,  and  test  equipment. 

J\'.i  /  •••!  I  I 

6.3  Topology  ^  :  \  j  I .  ] 

Fibre  Channel  defines  three  major  topologies  -  point-0-point,  fabric,  and  arbitrated  loop.  The 
point-to-point  topology  is  the  simplest.  If  connects  tWo  ports  With  a  bi-directional  link  (reference 
Figure  3).  I:  | .  I  r :  \  <v>/ 


Figure  3,  Point-to-Point  Topology 
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In  the  fabric  topology,  each  node  is  connected  to  a  switch.  Depending  on  the  capabilities  of  the 
switch,  any  node  may  connect  to  any  other  node  (reference  Figure  4).  When  denoting  fabric 
topologies,  the  fabric  is  shown  as  a  cloud.  This  represents  the  fabric  notion  without  showing  any 
physical  connections.  One  of  the  drawbacks  of  fabric,  is  the  requirement  for  one  or  more  fabric 
switches  that  physically  take  the  place  of  the  network  cloud.  These  are  not  necessarily  cheap  — 
especially  for  a  test  environment.  However,  because  of  the  connectivity,  adding  additional  nodes 
increases  the  total  bandwidth  available  to  the  system.  In  reality,  this  is  only  true  if  there  is  a 
broad  distribution  of  network  traffic.  If  all  nodes  are  trying  to  talk  through  one  link  to  the 
recorder,  then  more  nodes  will  only  make  it  worse. 


The  arbitrated  loop  topology  is  a  simple  concatenation  from  the  transmitter  of  one  node  to  the 
receiver  of  the  next.  This  progresses  through  all  nodes  until  the  last  one  is  connected  to  form  a 
loop  (reference  Figure  5).  Simplicity  its  one  of  the  advantages  of  a  loop.  There  is  no  additional 
network  hardware  required  for  connectivity^  To  add  more  nodes,  the  loop  is  broken  with  the 
additional  nodes  being  inserted  between  the  break.  One  of  the  drawbacks  of  a  loop  is  the 
constant  bandwidth.  Regardless  ofjtbe  number  of  nodes,  they  all  share  the  same  bandwidth. 


Figure  5,  Arbitrated  Loop  Topology 
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The  last  type  of  topology  available  is  the  hybrid  topology.  The  hybrid  topology  simply  replaces 
on  of  the  fabric  nodes  with  a  loop.  Conversely,  it  replaces  a  loop  node  with  a  fabric  (reference 
Figure  6).  This  is  one  instance  of  a  hybrid  topology,  of  which  there  are  many  variations.  This 
topology  has  the  pros  and  cons  of  both. 


6.4  Fault  Tolerance 

In  systems  most  instrumentation  engineers  are  familiar  with,  a  single  point  failure  rarely  brought 
the  system  to  its  knees.  With  traditional  instrumentetipn  ^tem^  aTaiilty  connection  on  a  data 
acquisition  unit  simply  meant  no  data  would  come  from  that  unit.  The  rest  of  the  system  would 
continue  to  operate.  The  same  is  truelfbr  Mi^Std-I553^4^S-[  With  switched  fabric  systems, 
the  switches  become  a  single  point  failure.  One  single-point-failure  mode  does  not  seem  like  a 
big  deal.  Current  systems  have  a  single  point  failure  in  the  system  controller.  When  we  consider 
arbitrated  loop  systems  -  each  node  on  the  loop  is  a  siiig'le-point-failure  source.  There  are 
several  ways  to  make  these  systems  more  fault  tolerant  such  as:  port  bypass  circuitry,  hubs,  and 
built  in  redundancy.  k-.-  'n  / 

6.4.1  Port  Bypass 

One  way  to  add  tolerance  to  a  loop  topology  is  to  add  port  bypass  circuitry  to  each  node.  If 
something  happens  to  the  node  (loss  of  power  or  other  problem)  the  bypass  kicks  in  and  allows 
the  loop  to  continue  to  operate.  The  node  designer  must  add  this  circuitry  to  the  unit  prior  to 
production.  The  port  bypass  circuit  will  not  help  a  faulty  connection  to  the  port  itself. 

6.4.2  Hub 

A  hub  allows  a  logical  loop  topology  to  be  physically  connected  in  a  star  fashion.  The  hub  acts 
as  a  security  guard  monitoring  the  health  of  each  of  the  ports.  When  it  detects  a  failure  on  one  of 
the  ports,  it  bypasses  the  faulty  port  within  the  hub  (reference  Figure  7).  In  this  way,  a  port  and 
its  associated  wiring  can  be  completely  removed  and  not  affect  the  system.  This  works  well, 
however,  many  of  the  drawbacks  of  the  switched  fabric  topology  have  been  reintroduced.  For 
example,  the  added  expense  (hardware  and  time)  of  routing  the  links  back  to  a  central  location  as 
well  as  the  cost  and  maintenance  of  the  hub. 
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Figure  7,  Arbitrated  Loop  with  Hub 


6.4.3  Redundancy 

Another  solution,  which  must  be  designed  into  the  port,  is  a  redundant  bus.  For  fabrics,  it  means 
multiple  ports  on  each  node.  Each  port  is  connected  to  the  fabric  and  receives  its  own  port 
address.  The  node  is  responsible  for  merging  data  from  among  its  ports.  |To  the  rest  of  the 
fabric,  it  looks  like  there  are  more  ports.  For  the  data  rates  expected  in  initial  instrumentation 
systems,  wholesale  redundant  busses  for  fabrics  do  not  seem  to  gain  mucKy.  However,  the 
concept  of  multiple  ports  for  high  bandwidth  data  sinks  like  recorders  has  merit.  For  loops,  an 
additional  connection  between  nodes  in  the  ppppsite.dirc^^uu  (is  -installed;  This  creates  a 

counter-rotating  ring.  If  there  is  a  connection  failure,  data  Can  still  traverse  the  ring. 

f  '  i  I  -  /  /;:■!/  f. 

.  ,  ^~V:  •••}  K;  f 

Avionics  busses  used  to  control  the  test  vehicles  have,  typically  had  redundancy  built  into  the 
system.  Given  the  bit  error  rates  of  operational  systerfts  in  the  past  and  the  criticality  of  a  failure, 
it  was  essential.  Redundancy  in  instrumentation  systems  has  been  the  exception  rather  than  the 
rule.  A  Fibre  Channel  system  built1  to  the  ANSf  standards  has  a  lower  bit  error  rate  than  anything 
used  previously.  The  system  designer  must  decide  if  redundancy  is  required  for  a  given 
implementation.  Possible  choices  include  counter  rotating  rings  and  dual  ported  nodes. 

6.4.4  Addressing 

When  a  port  logs  into  the  fabric,  or  when  the  loop  is  initialized,  the  port  addresses  are  assigned. 
Fibre  Channel  allows  a  port  to  request  a  previously  assigned  address.  It  suggests  [?]  the  port 
requests  an  address  on  a  cold  start.  The  big  concern  here  would  be  for  systems  where  new  nodes 
may  be  coming  online  at  random  or  under  some  other  control.  Since  the  test  vehicle  is  a  private 
system  where  the  instrumentation  engineer  has  the  knowledge  of  what  nodes  are  in  the  system, 
static  addresses  should  not  be  a  problem.  The  ability  to  preset  an  address  would  be  preferable  for 
many  reasons  -  not  the  least  of  which  would  be  trouble-shooting. 

6.5  Timing 

[We  need  to  do  some  work  here.  Given  the  tolerance  of  the  clock,  system  delays,  etc.,  what  is 
the  best  we  can  hope  for  empirically?  How  close  to  that  do  we  think  we  can  get? 

Network  Time  Protocol  (NTP)  ??? 
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[Loop  Timing 

Signaling  rate  of  1  062  Mbaud  ±100ppm  =  962  to  1 162  Mbaud 

Clock  rate  of  1.039  to  0.860  ns  =  ±0.09  ns 

Clock  uncertainty  from  126  nodes  =  ±0.09ns  *  126  =  ±11.3  ns 

If  propagation  delay  through  node  cannot  be  determined  precisely,  but  is  speced  ±10  ns, 
Additional  uncertainty  =  ±10ns  *  126  =  ±1 260  ns  =  ±1.26  us 

Even  if  the  node  is  speced  to  ±1ns,  it  would  still  add  ±126  ns  uncertainty  to  the  full  loop,  beyond 
our  original  100ns  requirement  (±100??)] 

Timing  is  one  of  the  most  critical  issues  facing  instrumentation  networks.  There  are  three  major 
timing  issues:  time  correlation  of  data,  simultaneous  sampling,  and  the  reconstruction  of  data 
sources.  Synchronizing  the  nodes  to  a  common  time  source,  if  done  accurately  enough,  could 
solve  all  three  issues.  Synchronization  issues  differ  upon  the  topology  selected. 


6.5.1  Data  Correlation 

Time  correlation  of  data  requires  knowledge  of  when  a  sample  occurred  ip  relation  to  other 
samples.  If  both  samples  occur  within  the  same  node,  the  issue  is  trivial:  When  they  occur 
across  different  nodes,  the  time  relationship  between  the  nodes  needs  to  be  known. 

6.5.2  Simultaneous  Sampling  j  |[ 

In  some  instances,  knowing  when  different  samples  occurred  is  not  good  enough.  The  samples 
need  to  be  acquired  at  the  same  moment  in  time  in  order  for  dipa:  processing  issues  to  be  reduced 
to  a  manageable  level.  ^  !  ‘  '  ■  '  ’  ! 


yf- 


Vi  / 


6.5.3  Data  Source  Reconstruction  f  ’  /  //  l 

Data  source  reconstruction  is  similar  to  data  correlation,  but  a  bit  more  specific.  For  some  data 
sources,  like  Mil-Std-1553  data  busses,  the  user  wants  to  recreate  the  bus  exactly  for  use  with 
simulators  or  trouble-shooting  equipment/  In  a  packet-based  environment,  each  packet  will  be 
stamped  with  the  time  of  arrival.  The  fidelity  of  the  time  stamps  will  vary  with  the  requirement 
for  reconstruction. 


6.6  Interoperability 

Section  4  discussed  what  the  interoperability  issues  were.  Section  5  was  the  interoperability 
implementation  requirements.  This  section  will  explain  some  of  the  rationale  of  why  certain 
values  were  selected. 


6.6.1  Cables  and  Connectors 

The  Fibre  Channel  standards  were  written  with  benign  environments  in  mind.  Because  of  space 
constraints  within  test  vehicles,  signal  wires  are  sometimes  tied  in  the  same  bundles  as  power 
lines  and  coaxial  cabling.  The  proximity  of  radars,  avionics,  and  power  distribution  units  creates 
an  environment  most  cable/connector  sets  cannot  tolerate.  Because  of  this  harsh  environment, 
the  physical  component  was  expected  to  deviate  from  the  standard.  Changing  the  physical  level 
should  not  affect  the  ability  to  leverage  the  commercial  industry. 
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6.6.2  Port  Type 

Since  this  is  an  interoperability  document,  it  was  decided  not  to  arbitrarily  choose  a  topology. 
There  are  pros  and  cons  to  both  that  the  system  designer  should  decide  what  is  best  for  their 
application.  The  selection  of  the  NLJPort  allows  any  of  the  topologies  to  be  used. 

6.6.3  Signaling  Rate 

For  two  nodes  to  communicate,  they  must  operate  at  the  same  signaling  rate.  Full  speed  is  by  far 
the  most  prevalent  and  the  one  most  vendors  will  design  into  their  units.  This  does  not  preclude 
the  use  of  other  rates  like  quarter  speed  or  faster  rates  in  the  future.  This  will  ensure  all  units 
have  a  common  rate  with  which  to  communicate. 

6.6.4  Login 

[Will  require  some  more  discussion/research... 

The  requirement  is  interoperability.  Since  the  instrumentation  network  is  a  private  network  - 
especially  in  the  beginning,  the  system  designer  knows  what  nodes  he  wants  to  put  on  the 
network  and  how  they  need  to  operate.  The  login  parameters  can  be  predefined  in  EEPROM 
or  something.  Explicit  login  seems  like  an  “auto-negotiate”  routine,  which  adds  a  level  of 
complication.  I  was  told  that  nodes  by  two  manufacturers  woutil  not  explicitly  login.  If  it 
wasn’t  for  implicit  logins,  they  wouldn’t  have  gotten  the  two  nodes  to  communicate.  I’ll  check 
for  more  details.  Probably  the  greater  concern  is  to  ensure  the  variety  of  login  parameters 
allow  interoperability.  For  example,  do  we  need  to  defirie  default  common  service  parameters 
for  FLOGI  and/or  PLOGI?  J  j 

6.6.5  Class  of  Service  '\  fi,.\  ^./[/}\  ''-j 

Much  the  same  as  signaling  rate,  Fibre  Channel  allows  several: choices.  However,  class  three 
seems  the  most  prevalent.  Again,  this  does  not  preclude  the'  use  of  other  classes. 

6.6.6  Protocol 

[This  needs  some  work  too.  I’m  leaning  towards  IP] 

Since  NexGenBus  did  not  study  the  upper  layer  protocols  (ULP),  selecting  the  most  capable 
protocol  is  out  of  the  question.  The  most  prevalent  ULP  seems  to  be  the  only  choice.  The  ULP 
used  frequently  on  Fibre  Channel  is  the  SCSI  protocol.  This  protocol  has  been  used  for  years  for 
read/write  commands  between  a  host  (PC)  and  a  target  (tape  drive).  Because  of  Fibre  Channel’s 
robust  architecture  and  low  latency  to  send  and  receive  SCSI  commands,  the  use  of  SCSI  in  a 
Storage  Area  Network  (SAN)  has  become  almost  universal.  Recently  the  use  of  TCP/IP  drivers 
on  Fibre  Channel  has  become  prevalent.  The  use  of  TCP  provides  the  ability  to  interoperate  with 
a  many  different  devices.  The  penalty  is  that  TCP  use  a  connection  oriented  protocol  in  which 
acknowledgments  are  received  for  each  packet.  This  creates  additional  traffic  on  the  network, 
which  reduces  throughput  and  increases  latency.  An  alternative  to  TCP  is  UDP,  which  uses  the 
same  size  packet,  etc.  but  does  not  acknowledge  packets  received.  This  increases  throughput  and 
decreases  latency.  Although  not  strictly  a  upper  layer  protocol,  the  Internet  Protocol  (IP)  is  the 
must  pervasive  protocol  in  use  today.  It  provides  a  connectionless  method  of  connecting  but  has 
a  rich  set  of  tools  developed  for  the  Internet.  The  IP  Protocol  is  used  with  either  TCP  or  UDP. 
Many  vendors  are  providing  IP  drivers  along  with  their  SCSI  drivers. 
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